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•  The  Challenge  of  Design  Under  Uncertainty 

•  Strategies  for  Considering  Uncertainty 

-  Epoch-Era  Analysis  (EEA) 

-  Modern  Portfolio  Theory  (MPT) 

•  Joint  EEA  and  MPT  Method  for  Affordability 

•  Case  Application:  Carrier  Strike  Group  (CSG) 
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THE  CHALLENGE 

OF  DESIGN  UNDER  UNCERTAINTY 
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Design  for  Value  Sustainment 


The  modern  warfighter  operates  in  a  global  environment  that  will 
inevitably  experience  dramatic,  dynamic  shifts  in  context 


Exogenous  uncertainties  exist  in  the  Stakeholder  needs  may  vary  with  the 

acquisition  and  operational  environment  decision  context 


-  Emerging  technologies  (e.g.,  UAS  maturation)  -  Change  of  stakeholder  preferences 

-  Political  transition  (e.g.,  low  carbon  fuels  mandate)  -  Change  of  mission  objectives 

-  Economic  shifts  (e.g.,  global  recession) 

-  Resource  availability  (e.g.,  rare-earths  crisis) 


Design  for  value  sustainment  assesses  system  performance  in  a 
variety  of  foreseeable  contexts  and  needs  during  conceptual  design 
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Design  for  Affordability 


•  74  Nunn-McCurdy  cost  breeches  between  1997  and  201 1 

•  Numerous  breeches  corresponded  to  context  changes  in  the 
environment  of  the  acquisition  programs<GAO’2011) 

•  A  variety  of  system-design  methodologies  have  been  developed  in 
response  to  the  Better  Buying  Power  (BBP)  mandates<Carter’  201°) 


http://www.navsource.org/archives  http://www.ainonline.com 


Can  systems  engineering  principles  create  sustained  lifecycle 
affordability  for  engineering  portfolios? 
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Design  Abstraction  Terminology 


Acquisition  and  development  efforts  face  different  challenges  and 
opportunities  contingent  on  the  scope  of  the  design  abstraction 


System-Level:  a  singular 
major  architectural  element 


Program-Level:  multiple  elements 
fulfilling  common  capability 
requirements 


Portfolio-Level:  multiple  elements 
that  collectively  fulfill  a  set  of  joint 
capabilities 
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Portfolio-Level  Design 


System-level  methodologies  do  not  effectively  enable  the  design  of 

specific  portfolio-level  properties 


•  Multi-system  acquisition  and  operation  of  portfolios  presents  higher 
order  complexities  not  addressed  by  system-level  design  techniques 

•  DoD  standards  for  SoS  design  are  described  in  the  Systems 
Engineering  Guide  for  System-of-Systems  (2008) 

•  Some  methods  have  also  been  adapted  for  portfolio  design 

-  Portfolio  Theory  application  for  SoS  decision  making<Davendralin9am  et  al'  2011> 

-  Real  options  analysis  for  IT  SoS  acquisition  strategies(Komoroski  et- al' 2006> 

-  T radespace-based  affordability  analysis  for  complex  systems(Wu  etal- 2014) 


Portfolio  design  for  lifecycle  value-sustainment  is  a  difficult  challenge 
requiring  advanced  systems  engineering  approaches 
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STRATEGIES  FOR 
CONSIDERING  UNCERTAINTY 
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Modern  Portfolio  Theory  (MPT)  for 
Engineering  Portfolios 


L 

Consistencies 


•  Value  elicitation  from  stakeholders 

•  Modeling  of  asset  value 

•  Founded  in  utility  theory 

•  Identifies  “efficient  frontier”  of 
potential  alternatives 


•  Asset  performance  is  non-Gaussian 

•  Portfolio  value  is  dictated  by  non-linear 
asset  performance  aggregation 

•  Covariance  is  insufficient  to  describe 
asset  correlation 

•  Asset  availability  is  dynamic 

•  Costs  may  accompany  diversification 


Select  elements  of  Modern  Portfolio  Theory  can  improve 
the  design  and  acquisition  of  engineering  systems  portfolios 
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Epoch-Era  Analysis 


Needs  (performance,  expectations) 


“ilities” 


Expectation  1 

Context 

1 


— 

Expectation  2 

Expectation  1 

Context 

Context 

2 

2 

Epoch  1  Epoch  2 


Epoch  3 


Short  run 
Long  run- 


Expectation  3 
NEW  NEED 
METRIC 


Context 

3 


Adding  a  new  system  to 
the  portfolio  may  meet 
changing  needs 


Portfolio  will  not  provide 
desired  utility  at 
expected  cost 


7/me 

(epochs) 


Epoch  4  Epoch  5 


+=Era 


Epoch 

A  time  period  of  static  contexts  and 
stakeholder  expectations,  like  a 
snapshot  of  a  potential  future 

Era 

An  ordered  sequence  of  epochs 
with  finite  time  durations;  a 
potential  progression  of  contexts 
over  the  portfolio  lifecycle 


Legend 

•  Portfolio 
—  ►  Portfolio  Trajectory 
Expectations 


(Ross  &  Rhodes,  2008) 


EEA  provides  a  method  to  compare  potential  portfolio  performance 
with  respect  to  the  dynamic  environment  in  which  they  operate 
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Multi-Attribute  Tradespace  Exploration 


•  Engineering  portfolio  design  has  traditionally  revolved  around  Analysis 
of  Alternatives  studies  concerning  a  few  promising  point  designs 

•  Multi-Attribute  Tradespace  Exploration  (MATE)  enables  designers  to 
consider  a  far  greater  set  of  alternatives  for  affordability<Wu  et  al-  2014> 


r 


Firm 

Designer 
Customer 
User 


Value  — ► 

Attributes 

H 

Utility 

* 

C! 

►  Analysis  - 

l  _ 

i 

Venables 


Tradespace  {DesignAttnbutes]  {Cost  Utility} 


Each  point  is  a 
specific  design 


The  combination  of  MPT,  EEA,  and  MATE  provides  new 
capability  for  portfolio-level  design  for  lifecycle  affordability 
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JOINT  EEA  AND  MPT  METHOD 
TO  SUPPORT  DESIGN  FOR  AFFORDABILITY 


mi  r 
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Portfolio-Level  Epoch-Era  Analysis 
for  Affordability  (PLEEAA) 
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Portfolio  Design  Tool 

•  Conducts  asset  allocation 

•  Applies  portfolio  class  constraints 

•  Enumerates  all  possible  portfolios 

Portfolio  Selector 

•  Compiles  a  specific  portfolio  for  modeling 


An  engineering  portfolio  may  be  represented  by  three 
primary  design  variables 

•  Legacy  Systems  -  existing  hardware  available  to  the  portfolio 

•  Acquisitions  -  new  assets  produced  for  the  portfolio 

•  Upgrades  -  change  options  available  for  legacy  systems 
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Portfolio  Design  Tool 


Fundamental  to  MPT,  asset  allocation  identifies  potential  classes 
of  assets  which  may  constitute  portfolio  elements 
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Constituent  System  Modeling 


Portfolio 

Value? 


www.public.navy.mil 


•  System-level  cost  attributes  are  directly  aggregated  to  portfolio  expense 

•  The  capability  tree  is  a  capability-based  value  mapping  to  aggregate 
system  performance  to  determine  portfolio  utility 
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Portfolio  Capability  Tree 


|  2nd  tier  of  hierarchy 
I  3^  tier  of  hierarchy 


Each  program  or  portfolio- 
level  manager  has  their  own 
model  to  assess  utility  from 
the  constituent  program  or 
svstem  utilities 


Portfolio 

cc-i  -  performance  attribute  1 
-  performance  attribute  2 

ctn  -  performance  attribute  n 


Each  “node"  entails  the 
ban  doff  of  information 
between  program  or 
portfolio-level  managers 


5 


Program  1 


Program  2 


Program  n 


SYSTEM-LEVEL  DESIGN!  VARIABLES 


Supports  top- 
down  transfer  of 
needs  and 
bottom-up 
aggregation  of 
performance 

Enables  the 
consideration  of 
complementary 
and  substitute 
system  impacts 
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MATE  with  Epoch-Era  Analysis 


Portfolio 

Utility 


Portfolio 

Expense 


\u 


.2! 

c 

o 

w 

c 

u 

E 

a 


d  o 


Each  point 
represents  a 
feasible  portfolio 


0  1 
Expense  (Dimensionless) 


Tradespace  of  Portfolios 

•  Utility  and  Expense  axes 

•  Multi-attribute  utility  theory  used  to 
describe  value  of  portfolio  performance 

•  Hundreds  of  thousands  of  portfolios 
may  be  visualized 


EEA  provides  several  techniques  to  analyze  the  promising  portfolio  designs 

•  Single-Epoch  Analysis:  identification  of  “promising”  portfolios  in  isolated  epochs 

•  Multi-Epoch  Analysis:  exploration  of  the  influence  of  contextual  uncertainty  on  a  set  of 
promising  portfolios 

•  Single-Era  Analysis:  identification  of  time-dependency  of  promising  portfolio  value  delivery 
through  multiple  epochs 

•  Multi-Era  Analysis:  exploration  of  path-dependency  of  promising  portfolio  value  delivery 
through  multiple  epochs 
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CASE  APPLICATION: 

CARRIER  STRIKE  GROUP  (CSG) 
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Portfolio-Level  Context  Definition 
and  Design  Formulation 


Identify  the  basic  problem  statement  and  design  space  for  the  proposed  portfolio 


VALUE  PROPOSITION 

“responsive,  flexible 
capability  for  sustained 
maritime  power  projection 
and  combat  survivability  to 
shape  the  operation 
environment,  respond  to 
crisis,  and  protect  the  US 
and  allied  interest  in  any 
threat  environment”  -  Chief 
of  Naval  Operations  (2010) 


PERFORMANCE 

ATTRIBUTES 

1 .  Electronic  warfare 
capability 

2.  Defensive  capability 

3.  Offensive  capability 

4.  Power  projection 

5.  Logistics 


Primary  portfolio  stakeholders 

-  Combatant  commander  (CCDR) 

-  Operational  commander 


EXPENSE 

ATTRIBUTES 

1. 

Acquisition  cost 

2. 

Influence  cost 

3. 

Operations  cost 

4. 

Schedule  cost 

Leaacv  Systems 

Potential  Constituent  Systems 
Leaacv  Systems  Acauisitions 

Uoarades 

Arleigh  Burke  Flight  1 

Nimitz  with  Complement 

Next  Generation  Combat  Ship 

Arleigh  Burke  Flight  1  upgrade 

Arleigh  Burke  Flight  II 

Los  Angles 

(NGCS)  -  6  variants 

Arleigh  Burke  Flight  II  upgrade 

Arleigh  Burke  Flight  IIA 
Ticonderoga 

Virginia 

Supply  Class 

Arleigh  Burke  Flight  IIA  Restart 
Arleigh  Burke  Flight  III 
Zumwalt 
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CSG  Capability  Tree  Formulation 


Portfolio  Level  Capability  Attributes 
2nd  Level  SoS  Capability  Attributes _ 

3rd  Level  SoS  Capability  Attributes _ 

ft  of  System  Level  Performance  Attributes 


Levels  of  capability  tree 
hierarchy 


CSG  Electronic  Warfare  Capability 

Early  Warning 

6 

Weapon  system  detection 

6 

Electromagnetic  System 

5 

CSG  Defensive 
Capability 

Battlespace  Defense 
Capability 

Sea  superiority 

5 

Air  Superiority 

5 

Undersea  Superiority 

5 

Combat  Search  and  Rescue 

2 

Naval  Asset  Defense 
Capability 

Anti-Ship  Missile  Defense 

5 

Anti-Ship  Kinetic  Weapon 

5 

Sea  Mine  Defense 

5 

Torpedo  Defense 

5 

Crew  Defense 

4 

Naval  Gun  Fire  Support 

5 

CSG  Offensive 

Missile  Strike 
Capability 

Ballistic  Missile  Interception 

5 

Cruise  Missile  Strike 

5 

Capability 

Torpedo  Capability 

5 

Sea  Basing  Capability 

3 

Special  Forces  Insertion 

4 

2 

« 1 

Power  Projection 

1  CSG  Logistics 

Branch  of 
capability  tree 
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CSG  Epoch  Characterization 


Seven  epoch  variables  identified  yielding  a  total  of  2187  distinct  epochs 


EV  Category 

Epoch  Variable 

[Range] 

Units 

EV  -  Technology 

Advanced  Energy  Weapons  (AEW) 

[0,  5,  40] 

MW 

EV  -  Technology 

Unmanned  Aerial  Systems  (UAS) 

[0,  2,  5] 

Berths 

EV  -  Maintenance 

Overhaul  Event  Costs 

[0,  0.5e9,  2e9] 

Billions  $ 

EV  -  Policy 

Budget 

[80,  100,  150] 

% 

EV  -  SoS  management 

Cooperation  Costs 

[80,  100,  150] 

% 

EV  -  Threats 

Enemy  Threat 

[Low,  Med,  High] 

Level 

EV  -  Threats 

Asymmetric  Threat 

[Low,  Med,  High] 

Level 

Five  epochs  initially  selected  for  demonstration  through  the  Carrier  Strike  Group  case  study 


Epoch  Names 

AEW 

UAS 

Overhaul 

Epoch  Variables 

Budget  Cooperation 

Enemy 

Asymmetric 

Baseline 

0 

0 

0 

100 

100 

Low 

Med 

Small  Navy 

0 

2 

0 

80 

150 

Low 

Low 

War  on  Terror 

5 

5 

0 

100 

80 

Low 

High 

Major  Conflict 

40 

5 

0 

150 

80 

High 

Med 

Peacekeeping 

5 

0 

0.5e9 

100 

100 

Med 

Med 
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Design-Epoch-Era  Tradespace 

Evaluation 


•  Based  upon  the  19  potential  constituent  systems 

-  53,108,336  unique  portfolios  were  enumerated 

-  524,160  portfolios  were  evaluated 

-  Between  220  and  477,916  portfolios  were  valid,  depending 
upon  the  epoch 


/ 


/ 


/ 


/ 


Epoch 

Valid  Portfolios 

Yield 

Baseline _ 

_ 173,581 _ 

_ 33,1% _ 

Small  Navy 

220 

0.04% 

War  on  Terror 

140,398 

26.8% 

Major  Conflict 

477,916 

91.2% 

Peacekeeping 

191,558 

36.5% 

I 


~  s 


s 


s 


s 


\ 


Severely  limiting  epoch  due  to  a  20%  budget  cut  and  50%  rise  in  cooperation  costs 


The  PLEEAA  method  enables  a  designer  to  consider  far  more 
alternatives,  each  in  numerous  potential  future  scenarios 
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Single-Epoch  Analysis 


Tradespace  Exploration  is  conducted  independently  in  each  epoch 


0.5  0.6  0  7  0.8  0.9  l 

MAE 


07 

06 

- 

05 

04 

03 

-A 

0* 

02 

— 

0.1 

— 

n 

# 

iiat; 

iLAE 

Portfolio  Composition 

A 

0.2651 

0.0352 

1  Arleigh  Burke  Flight  III 

1  Nimitz  with  Complement 

I  Los  Angles 

B 

0.4109 

0.1487 

1  Arleigh  Burke  Flight  I 

1  Arleigh  Burke  Flight  II 

1  Arleigh  Burke  Flight  III 

1  Nimitz  with  Complement 

1  Los  Angles 

C 

0.4800 

0.1953 

2  Arleigh  Burke  Flight  III 

1  Nimitz  with  Complement 

1  Los  Angles 

D 

0.6305 

0.3920 

2  Arleigh  B.  Flight  II  Upgrade 

2  Arleigh  Burke  Flight  III 

1  Nimitz  with  Complement 

1  Los  Angles 

E 

0.6532 

0.6206 

2  NGCS  Variant  6 

2  Arleigh  B.  Flight  II  Upgrade 

2  Arleigh  Burke  Flight  III 

1  Nimitz  with  Complement 

1  Virginia 

Promising  portfolios  are  identified  on  the  Pareto  frontier  of  each  epoch 
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Multi-Epoch  Analysis 


Promising  portfolio  designs  are  simultaneously  explored  in  multiple  epochs 


Baseline 


War  on  Terror  Major  Conflict 


0  0.1  0,2  0.3  0  4  0.5  0.6  0  7  0. 

MAE 


MAE 


Multi-Epoch  analysis  illustrates  the  influence  of  contextual  uncertainty 
on  the  utility  of  potential  Carrier  Strike  Group  portfolios 
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Multi-Epoch  Analysis 


Promising  portfolio  designs  are  simultaneously  explored  in  multiple  epochs 

Small  Navy  Epoch 


Baseline 


0.7 

06 

0.5 

04 

0.3 

0.2 

0.1 


Major  Conflict 


Multi-Epoch  analysis  illustrates  the  influence  of  contextual  uncertainty 
on  the  utility  of  potential  Carrier  Strike  Group  portfolios 
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CSG  Era  Construction 


•  An  era  is  an  ordered  sequence  of  epochs 

•  Evaluating  portfolio  designs  over  an  era  illustrates  the  potential 
lifecycle  value  robustness  of  the  portfolio 

•  Two  eras  were  constructed  from  the  five  epochs  through  a 
narrative  approach 


ERA  1 

Baseline  (5yr)  — ►  War  on  Terror  (5yr)  — ►  Peacekeeping  (lOyr)  — ►  Baseline  (3yr)  — ►  Small  Navy  (7yr) 


TIME 


ERA  2 

Peacekeep.  (5yr)  — ►  Small  Navy  (5yr)  — ►  Major  Conflict  (5yr)  — ►  Peacekeep.  (12yr)  — ►  Baseline  (3yr) 
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Single-Era  Analysis 


Promising  portfolio  designs  independently  explored  in  the  constructed  eras 


Baseline  (5yr)  — ►  War  on  Terror  (5yr)  — ►  Peacekeeping  (lOyr)  — ►  Baseline  (3yr)  — ►  Small  Navy  (7yr) 


EPOCH  5  yr  BL  5  yr  WoT  lOyrPK  3yrBL  7yrSN  EPOCH  5  yr  BL  SyrWoT  lOyrPK  3yrBL  7yrSN 

< - >  < - > 


ERA 


Legend  ♦ABB  C  V  D  E 


ERA 


Single-Era  Analysis  enables  exploration  of  the  time-dependent 
affordability  of  promising  CSG  portfolios  in  one  potential  future 


seari.mit.edu 


May  13-14,  2015 


28 


SF/V 


Discussion 


Systems  Engineering 
Research  Initiative 


Advancement 


•  The  PLEEAA  method  supports  design  for  affordability  during 
conceptual  design 

-  Considers  new  contexts  before  they  arrive 

-  Assesses  the  lifecycle  value  sustainment  of  potential  portfolios 

-  Communicates  portfolio  values  to  constituent  systems 

-  Aggregates  constituent  system  performance  to  portfolio  utility 

•  The  case  study  enables  acquisitions  officers  and  designers  to  explore 
promising  CSG  portfolio  performance  in  numerous  potential  futures 
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System,  Program  and  Portfolio 


Acquisition  and  development  efforts  face  different  challenges  and 
opportunities  contingent  on  the  scope  of  the  design  abstraction 


System-Level:  Design  that  is  inclusive  of  a  singular 
major  architectural  element  that  is  semi-independent 
from  the  remainder  of  the  architecture 


Program-Level:  Design  that  requires  joint  consideration 
of  multiple  independent  or  semi-independent 
constituent  elements  such  that  each  element  fulfills  a 
common  set  of  capability  requirements  subject  to 
identical  stakeholder  value  metrics 


Portfolio-Level:  Design  that  seeks  to  create  a  collection 
of  heterogeneous  assets,  both  from  legacy  and  new 
sources,  that  can  collectively  provide  a  set  of  emergent 
capabilities  through  the  aggregate  performance  of  each 
constituent  system 


www.public.navy.mil 
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Modern  Portfolio  Theory 


•  Utilized  by  financial  institutions  and 
operations  research  since  the  1950’s 

•  Constructs  groupings  of  investments  that 
maximize  return  (utility)  subject  to  an 
acceptable  threshold  of  risk  (cost) 

•  Result  in  an  “efficient  frontier”  of  potential 
investment  sets 

•  Relies  upon  negative  trending  covariance  in 
diversified  assets  to  reduce  aggregate  risk,  or 
Mean-Variance  optimization 

•  A  variety  of  MPT  derivatives  exist  which 
introduce  non-normally  distributed  risks  and 
semi-variance  among  assets 
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Complementary  and 
Substitute  Systems 


http://www.navsource.org/archives 


Complementary  Systems 

•  Value  delivery  enhanced  in  at  least 
one  performance  attribute 

•  Gain  new  capability  in  a  performance 
attribute 

•  Often  results  from  a  change  to  the 
system’s  CONOPS 


http://www.navy.mil/navydata 


Substitute  Systems 

•  Simultaneous,  overlapping  value 
delivery  in  a  performance  attribute 

•  Often  dependent  upon  the  CONOPS 

•  Systems  may  be  substitute  in  one 
performance  attribute,  but  not 
necessarily  in  others 


PLEEAA,  provides  two  mechanisms  to  address  complementary  and 
substitute  systems  through  the  capability  tree  architecture 
1.SME  matching  with  potential  interaction  opportunities 
2. Level  of  Combination  Complexity  adjustment  factors  (Chattopadhyay,  2009) 
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Case  Study  Application  of  PLEEAA 


seari.mit.edu 


May  13-14,  2015 


36 


Future  Work 


Rl 


IMif 


Systems  Engineering  Advancement 
Research  Initiative 


•  The  work  conducted  in  this  research  represent  initial  efforts  to 
extend  EEA  to  the  portfolio-level  of  design 

•  Numerous  opportunities  exist  to  improve  PLEEAA  techniques,  and 
add  additional  capabilities 

-  Expanded  schedule  cost  factors 

-  Dynamic  entry  and  exit  of  systems  from  portfolios 

-  More  extensive  collaboration  costs  and  “likelihood  of  participation”  factors 

-  Design  for  “graceful  degradation”  capability 

-  Expanded  mechanism  to  characterize  complementary  and  substitute  systems 
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